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Abstract 

We studied the computational properties of an attractor neural network (ANN) 
with different network topologies. Though fully connected neural networks exhibit, 
in general, a good performance, they are biologically unrealistic, as it is unlikely 
that natural evolution leads to such a large connectivity. We demonstrate that, 
at finite temperature, the capacity to store and retrieve binary patterns is higher 
for ANN with scale-free (SF) topology than for highly random-diluted Hopfield 
networks with the same number of synapses. We also show that, at zero temperature, 
the relative performance of the SF network increases with increasing values of the 
distribution power-law exponent. Some consequences and possible applications of 
our findings are discussed. 
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1 Introduction 



There is a growing interest in evolving complex networks, in particular, net- 
works with scale-free (SF) topology [1-3]. SF occurs in many different con- 
texts, including the WWW and the Internet, e-mail and scientific-citation 
networks, ecological, protein and gene interaction networks, etc. In these ex- 
amples, the degree k of a vertex, i.e., the number of arcs linking it to other 
vertices, is power-law distributed, P(k) ~ A; -7 (see figure 1). This implies that 
the network includes a relatively large number of nodes with small connectiv- 
ity, denning what we call the network boundary, and a few nodes, the hubs, 
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with a large connectivity, comparable to the network size N. As a consequence, 
SF networks exhibit the interesting small-world property, that is, the average 
path length between two nodes is very small compared to the network size. 

Evolving networks with such complex topology are also common in biology. 
Neuronal networks, for instance, seem to exhibit the small-world property. 
This was recently demonstrated in a set of in-vitro experiments of growing 
cultured neurons [4]. Although an impressive amount of work has been done 
in the last few years concerning SF networks, it has only recently been re- 
ported on the specific consequences of such an architecture on the performance 
of auto-associative neural networks [5,6]. The authors in [5] show that a SF 
neural network is able to store and retrieve P patterns with a lower computer- 
memory cost than the fully-connected Hopfield neural network. They also find 
a similar performance with a (biologically unrealistic) nearest-neighbor hyper- 
cubic Ising lattice. The authors in [6] study the zero temperature behavior of 
different topologies, namely, the Barabasi-Albert (BA) SF, small-world and 
random diluted networks, and a better performance for the random diluted 
case than for the other topologies is reported. However, for the relative large 
mean connectivity these authors use ((k) = 50), the BA network has not the 
SF property [5], so that this result lacks interest. 

We here report on the influence of topology on the associative—memory task of 
the network as a function of temperature. In particular we focus on two main 
issues, namely, on the robustness of the network performance against thermal 
noise for varying topology, and on the effect of varying the SF connectivity 
distribution P(k) on the network performance. 



2 Definition of Models 

Consider the BA evolving network [1] with N nodes and r](N — i]q) links. 
Here, rjo is the initial number of nodes generating the network, n < rjo is the 
number of links that are added during the evolution at each time step, and 
iV is the final number of nodes in the network. This will latter be generalized 
to consider other SF networks. In order to have a neural system with the 
chosen topology, we place a binary neuron, s« = 1 or 0, at each node i, and 
then "store" P binary random patterns, ^ = {£f = 1 or 0} , fi = 1, . . . , P, 
with mean activity level = 1/2. This is done in practice by associating a 
synaptic intensity Wj 3 - at each link according to the Hebbian learning rule, 
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Fig. 1. Left: Connectivity probability distributions for a SF network with 7 = 3 
(filled circles) and for a HDH network (open circles). Right: Connectivity probability 
distributions generated for a SF network by tuning the exponent 7. Each data point 
corresponds to an average over 100 networks with N = 4000 neurons each. 

A meaningful direct comparison of this SF neural network (SFNN) and the 
standard Hopfield neural network cannot be made because the second case 
involves N 2 synaptic connections. A hypercubic Ising lattice has the same 
number of synapses than the SFNN; however, real neural systems are known 
to exhibit more complex neuron connectivity than the Ising network. Con- 
sequently, we compare the performance of the SFNN with that of a highly 
diluted Hopfield network (HDHN). The HDHN is obtained from the standard 
Hopfield network by randomly suppressing synapses until only r)(N — r) ) of 
them remain, i.e., the number of synapses scales as N and not as N 2 . To 
maintain the SF behavior in the BA network, the value of rj must be very 
small compared to the network size, that is, rj <C iV [5]. The connectivity 
distribution of the HDHN is illustrated in figure 1 (left). The main differences 
between this distribution and the corresponding one for a SF network is that 
the latter has no typical connectivity value. More specifically, the SF network 
distribution is a power-law while the HDHN distribution has a maximum and 
an Gaussian decay and, consequently, may be characterized by a (typical) 
mean connectivity. 

A relevant magnitude to monitor in order to compare the performance of 
different topologies is the overlap function, denned for pattern v as 
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The performance of the two networks is compared in figure 2 for P = 1 and 
77 = 3. This clearly shows that, excluding very low temperature, the retrieval of 
information as characterized by m v is better for the SFNN than for the HDHN. 
In both cases, the retrieval of information deteriorates as P is increased. How- 
ever, we also observe that, at finite temperature, the performance of the SFNN 
increases significantly if one considers only the retrieval of information con- 
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Fig. 2. Right: Overlap curves for varying temperature (in arbitrary units) for a BA 
network (filled squares) and a HDHN (filled circles). Data correspond to an average 
over 100 realizations of a network with iV = 1600 neurons, P = 1 and 77 = 770 = 3. 
Left: Local overlap m(k) for a BA network as a function of the connectivity k for 
increasing values of the temperature (successive decreasing curves). 

cerning neurons with a connectivity degree higher than certain value, ko, i.e., 
the hubs [7]. This can be understood on simple grounds by computing the local 
overlap m\ with one pattern for neuron i and averaging over all neurons with 
the same connectivity k. The resulting mean overlap for a given connectivity 
m(k) is plotted in figure 2 (right) for different temperatures. This shows that, 
even at high temperature, the local overlap for hubs is close to one whereas it 
is very small for the boundary (fluctuations in the lower curves are due to the 
small number of hubs present in the network for the relative small network size 
we are using). This finding reflects the "negative" role of the boundary and 
the "positive" role of the hubs on the SFNN performance during each retrieval 
experiment when thermal fluctuations are considered. This observation is in 
agreement with the T = behavior reported in [6]. 

Another important issue is how the exponent of the distribution influences 
the performance of SF network for associative memory tasks. In order to ana- 
lyze this, we studied networks characterized by different power-law exponents. 
With this end, we used a Molloy-Red (MR) SF network [8] with P(k) ~ A;" 7 , 
where 7 is a tunable parameter. As illustrated in figure 1 (right), the number 
of neurons in the network with a high connectivity increases as 7 is decreased. 

Even more interesting is when one compares the behavior of the MR network 
with that of the HDHN in the limit T = (cf. figure 3). As thermal fluctua- 
tions are then suppressed, the network performance is only perturbed by the 
interference among the stored patterns. In order to consider the limit of inter- 
est, we started with one of the stored patterns, and computed the state of each 
neuron according to the deterministic rule Si(t + 1) = Q(hi(t)). Here, Q(x) is 
the Heaviside step function, and hi = J2) w ij s j is the local field associated 
with neuron % with the sum over all neurons j connected to it. At the end 
of the network evolution, we recorded the overlap, mf ina i, with the starting 
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Fig. 3. Left: Zero temperature performance of a Molloy-Red SFN for 7 = 1.5, 2, 2.5, 3 
(solid lines) compared to the performance of different HDHN with the same number 
of synapses (dashed lines), and as a function of the number of stored patterns P. 
Right: Relative difference in the performance for the cases showed in the left panel. 



pattern as a function of the total number of stored patterns P. In order to 
visualize the difference in performance between the two types of networks, we 
defined Amf ina i = m,f^ N — mf^^ 1 ^ . Figure 3 shows how this difference in 
performance varies with the number of stored patterns. The graph illustrates 
that the SFNN has a better and better performance as compared with the 
HDHN as the number of stored patterns is increased. This effect is enlarged 
as 7 is increased. This can be understood by considering the different decays 
of P(k) for large k in both SFNN and HDHN, and the fact that for increasing 
values of 7, the relative position of (k) moves to the left for both topologies, 
but due to the power-law decay the effect of the hubs remains for the SFNN. 



Summing up, the topology of a neural network has a key role in the processes 
of memorization and retrieval of patterns. In particular, neural networks with 
scale-free topology may exhibit a better performance than Hopfield-like net- 
works with the same number of synapses distributed randomly over the net- 
work. Our study can be useful to understand the role of regions with different 
connectivity degrees in real neural systems during memorization and retrieval 
of information. In particular, it may improve our understanding of how fluctu- 
ations or perturbations in the typical number of synapses of some brain areas 
can affect the processing of information and memorization in these regions. 
Our study also suggests the convenience of developing new methods to store 
the more relevant information into the hubs, increasing in this way the effec- 
tive network-performance and efficiency. It would be desirable to check our 
findings against experimental observations on real neural systems focusing on 
the topology which is built up by natural selection. 



Acknowledgments 



We tanks Dr. Pastor-Satorras for fruitful suggestions. This work was supported 
by the Spanish MCyT and FEDER "Ramon y Cajal" contract and project no 
BFM2001-2841. 



References 



[1] R. Albert, A. Barabasi, Statistical mechanis of complex networks, Rev. Mod. 
Phys. 74 (1) (2002) 47-97. 

[2] S. Dorogovtsev, J. Mendes, Evolution of networks, Adv. Phys. 51 (2002) 1079- 
1187. 

[3] R. Pastor-Satorras, A. Vespignani, Evolution and Structure of the Internet, 
Cambridge University Press, Cambridge, 2003. 

[4] O. Shefi, I. Golding, R. Segev, E. Ben- Jacob, A. Ayali, Morphological 
characterization of in-vitro neuronal networks, Phys Rev E 66 (2) (2002) 021905. 

[5] D. Stauffer, A. Aharony, L. Costa, J. Adler, Efficient hopfield pattern recognition 
on a scale-free neural network, Eur. Phys. J. B 32 (3) (2003) 395-399. 

[6] P. McGraw, M. Mezinger, Topology and computational performance of atractor 
neural networks, cond-mat/0304021. 

[7] J. Torres, M. Munoz, J. Marro, P. Garrido, to be published. 

[8] M. Molloy, B. Reed, A critical-point for random graphs with a given degree 
sequence, Random Struct. Algorithms 6 (2-3) (1995) 161-179. 



6 



